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Does Mechanism Matter? Student Recall of Electronic versus 
Handwritten Feedback 


Abstract 

Student consumption and recall of feedback are necessary preconditions of successful formative assessment. 
Drawing on Sadler s (1998) definition of formative assessment as that which is intended to accelerate learning 
and improve performance through the providing of feedback, we examine how the mechanism of 
transmission may impact student retention of feedback content. We proceed from the premise that such 
retention is necessary for feedback to function as a component of formative assessment. Although researchers 
have written extensively on best practices in feedback content (e.g., Nicol & Macfarlane-Dick, 2006) and 
student and instructor attitudes toward electronic feedback versus handwritten feedback (e.g., Thomson, 
2008), comparatively little research addresses whether the form of feedback influences student consumption 
and retention. Our research found that whereas students who preferred or received handwritten feedback 
recall more feedback (quantity), those who actually received electronic feedback recall comments more 
accurately (quality). We encourage instructors to working with either format to adhere to accepted standards 
for good feedback practice. 
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Does Mechanism Matter? Student Recall of Electronic versus 

Handwritten Feedback 

As higher education has increasingly embraced online 
learning and integrated technology into course deliverables for 
traditional classroom courses, assessing the impact of electronic 
tools on learning outcomes has become essential. Of particular 
interest has been how such e-learning tools mitigate the 
relationship between learner and educator as well as how such 
tools can be used to best facilitate learning. Prior research has 
often focused on student preferences regarding assessment 
transmission (Bridge & Appleyard, 2008; Brodie & Lock, 2009; 
Thomson, 2008); the current research moves forward to 
examine student retention of feedback based on whether it takes 
the form of e-feedback or handwritten paper-based assessment 
in a traditional classroom context. The value here is in directly 
assessing the extent to which students consume feedback, 
rather than assuming that student preferences regarding 
feedback mechanism will necessarily lead to more considered 
consumption. 

Feedback, which includes grades and comments provided 
by an instructor to a learner, is central to higher education, as 
quality feedback facilitates effective student learning (Bransford 
et al., 2000; Carless et al., 2011; Chickering & Ehrmann, 2008; 
Chickering & Gamson, 1991; Espasa & Meneses, 2010; Gikani et 
al., 2011). The "first two broad functions of feedback are to 
provide the teacher's assessment of the student response, 
maybe as a grade, with a rationale that indicates how the 
judgment took the strengths and weaknesses of the response 
into account. The second function is to provide advice or 
suggestions as to how a better response could have been 
constructed" (Sadler, 2009, p. 3). Defined as an evaluation of 
learner achievement or progress toward a learning objective, 
assessment thus forms the core of feedback content (Gikani et 
al., 2011; Reeves & Hedberg, 2009). Further, researchers (e.g., 
Challis, 2005; Oosterhof et al., 2008) often subdivide 
assessment into two fundamental types, summative and 
formative. 

A common form of summative assessment is the grade an 
instructor assigns a student to indicate the extent to which the 


https://doi.org/ 1 0.2042 9/ij SO tl.20 1 5.090 107 


1 



Does Mechanism Matter? 


student has achieved a learning objective associated with a 
particular assignment, module, semester, or the end of some 
other instructional unit (Challis 2005; Gikandi et al., 2011; 
Hargreaves, 2008). In contrast, the purpose of formative 
assessment is to improve student learning and future student 
performance on a learning objective that is not considered 
complete at the time of the assessment (Nicol & Milligan, 2006; 
Sadler, 1998). Formative assessment is "specifically intended to 
provide feedback on performance to improve and accelerate 
learning" (Sadler, 1998, p. 77). Formative feedback is an 
essential course deliverable with which students can evaluate 
their conception of course goals and standards and become 
empowered as self-regulated learners (Nicol & Macfarlane-Dick, 
2006; Pintrich & Zusho, 2002). 

As instructors move away from paper copies of written 
work, both summative as well as formative assessment have 
moved toward electronic feedback delivery. Less clear in the 
existing literature is the extent to which the use of technology to 
provide feedback may impact the extent to which feedback 
intended to be formative may become summative in the hands 
of the student. Formative feedback becomes summative in this 
way when students review only the gradebook in a learning 
management system but fail to open any comments, rubric, or 
instructor edited version of their work, for example. That is, 
formative assessment cannot be exclusively the purview of the 
instructor but must also be engaged by the learner, ultimately 
promoting self-regulated learning (Espasa & Meneses, 2010). 
Therefore, formative assessment was the goal of the feedback in 
the current research. 

The transition to self-regulated learning, regardless of 
transmission format, can only occur when clear articulation of 
goals is coupled with quality formative feedback (Black & 
William, 1998; Garcia, 1995; Nicol & Milligan, 2006; Sadler, 
1989). There are accepted standards for good feedback practice 
that bear mentioning (adapted from SENLEF, Nicol & Milligan, 
2006, Pintrich & Zusho, 2002): 

1) Helps clarify what good performance is (goals, criteria, 
expected standards) 


https://doi.org/! 0.20429/ijsotl.2015.090107 


2 



IJ-SoTL, Vol. 9 [ 2015 ], No. 1 , Art. 7 


2) Facilitates the development of reflection and self- 
assessment in learning (Black & William, 1998; Sadler, 1998) 

3) Delivers high quality information to students about their 
learning 

4) Encourages teacher and peer dialogue about learning 
(Freeman & Lewis, 1998) 

5) Encourages positive motivational beliefs and self-esteem 

6) Provides opportunities to close the gap between current 
and desired performance 

7) Provides information to teachers that can be used to shape 
teaching 

Hatziapostolou and Paraskakis (2010, p. Ill) add that formative 
feedback must also "be timely, constructive, motivational, 
personal, manageable and directly related to assessment criteria 
and learning outcomes." 

Providing high quality feedback may facilitate learning but 
can only do so if learners engage with the feedback content 
rather than merely observing the grade. Even the highest quality 
feedback cannot engender changes in student performance if 
students do not review the content of the comments; we 
therefore must consider not only what constitutes good feedback 
but also how to transmit feedback in such a way as to ensure 
that students encode the content. 

Adams (2011) points out the current lacuna in research 
examining potential disparities in feedback consumption and 
engagement based on electronic versus hard copy dissemination. 
Prior research comparing handwritten to electronic feedback is 
limited in that it often focuses primarily on student preferences 
(Bridge & Appleyard, 2008; Brodie & Lock, 2009; Thomson, 
2008). Student preferences regarding feedback mechanism are 
inconsistent across studies, and some differences may be 
attributed to features of the research beyond the difference 
between e-feedback and traditional assessment. 

In the context of distance learning, students preferred 
electronic feedback as easier to read and easier to reference at a 
later time but also expressed preference for e-feedback as a 
time savings over mailed submission and feedback. Despite the 
generally positive view of e-feedback, students in Thomson's 
(2008) study also expressed concern over difficulties with the e- 
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feedback software and/or personal computing software. 

Thomson herself notes that "the e-system has the potential of 
some formatting being 'lost in translation'...on the other hand, of 
course, the system removes the students' struggles with illegible 
handwriting" (2008, p. 5). 

Bridge and Appleyard (2008) also examined preferences 
for e-feedback over traditional assessment in the context of 
distance learning and found that 55% of students preferred or 
strongly preferred online feedback and made comments similar 
to those noted by Thomson (i.e., they valued the reduced time 
lag between submission and feedback but also viewed e- 
feedback as impersonal). Further, Brodie and Loch (2009) found 
that students and instructors preferred handwritten comments 
due to their more personal and specific nature. 

While preferences are important to the learning process, 
we should not assume that preferences are proxy for the level of 
attention students give to feedback. Attention to feedback is 
necessary for feedback to function formatively rather than 
merely as summative assessment. Acknowledging this, we 
examine student recall of feedback for both e-feedback and 
handwritten feedback in the traditional classroom learning 
environment. We explore in our study not only student 
preferences regarding feedback mechanism, but also the 
quantity and quality of their recall of specific feedback. 

Method 

Participants 

A total of 34 male and female undergraduate students 
participated in this study. Thirteen were enrolled in a Political 
Science course (Politics of Gender and Sexuality), and 21 were 
enrolled in Psychology courses (10 in Advanced Research, 11 in 
Health Psychology). All three courses were at the 2000- or 
sophomore-level, though students ranged from freshmen to 
seniors. A wide range of majors were represented. Nineteen 
students received written feedback, and 15 received electronic 
feedback. 

Materials and Procedures 

Each course was taught by a different instructor (the three 
authors of the current research), and the writing assignments 
varied. For the Politics of Gender and Sexuality course, students 
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analyzed three authors' arguments in a critical review of the 
assigned readings. For the Advanced Research in Psychology 
course, students submitted a literature review for an 
independent, empirical research project. Finally, for the Health 
Psychology course, students submitted a progress report for a 
semester-long health promotion project. All students submitted 
their papers electronically via a learning management system 
(Blackboard) and were randomly selected to receive their 
feedback either handwritten or electronically. We made an effort 
to keep our comment style and length of comments consistent 
between formats but did not attempt to make feedback 
consistent across instructors. Instead, we maintained our unique 
feedback styles while manipulating feedback mechanism. 
Students were not aware of the varied format of feedback, and 
we kept a copy of the actual feedback for data purposes. 

Students were encouraged to read their feedback but were 
not made aware of the feedback retention survey as this would 
likely influence their attention to the feedback and thus skew our 
results. Because prior consent would influence the results, all 
students were given the opportunity to remove their data from 
the study. The institutional review board accepted these 
methods for testing on human subjects in this instance. 

Feedback retention survey. We gave students the 
feedback (whether written and electronic) at one class period 
and then surveyed the students about the feedback at the next 
class meeting. For each course, there was one day between 
receiving feedback and completing the survey. On the survey, 
students received the following instructions: Please report any 
and all feedback I gave you on your paper. Be as specific and 
detailed as possible. Use my exact words when you can. 

Students also indicated how closely they read the 
feedback, on a scale of 1 (not at all) to 5 (very carefully). 

Finally, they indicated which type of feedback they preferred 
(written or electronic) and why. 

Debriefing. After collecting the surveys, we debriefed the 
students, who could opt out if they did not want their data to be 
used. All students agreed to allow their data to remain in the 
dataset. 
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Scoring. In a series of meetings, we each scored every 
survey according to the metrics below, and we resolved 
discrepancies by consensus. 

Instructor Total. First, we discussed and reached 
agreement on what constituted a discrete comment. We then 
counted the number of discrete comments the instructor made 
on each student paper. For example, "you seem to be skipping 
over some vital data" counted as one discrete comment, 
whereas, "Good comparison between the articles. You need more 
synthesis though" counted as two. 

Student Total. Following a similar process, we next went 
through the students' feedback retention surveys and counted 
the total number of comments they recalled (correctly or 
incorrectly). 

Correctly Reported Comments. Finally, we calculated a 
Correctly Reported Comments by counting the number of only 
correctly recalled comments. 

From these three variables, we calculated three compound 
variables for further analysis: 

Quantity. This variable is the ratio of Student Total to 
Instructor Total. 

Encoding. This variable is the ratio of Accurately Reported 
Comments to Instructor Total. 

Accuracy (Quality). This variable is the ratio of Correctly 
Reported Comments to Student Total. 

Results 

Analysis Methods 

We test for relationships between the aforementioned 
variables using chi-squared, a test of statistical significance for 
relationships between nominal variables. Because chi-squared is 
comparing the observed association between nominal variables 
to the expect relationship were there no relationship, the 
underlying premise is that chi-squared tells us the likelihood that 
our observed relationship is not merely by chance. 

For some models it was necessary to adjust for 
heteroscedasticity. Heteroscedasticity occurs when the variation 
our dependent variable (see models below for specific dependent 
variables) have an unequal spread across the range of the 
independent variable. 


https://doi.Org/l 0.20429/ijsotl.2015.090107 


6 



IJ-SoTL, Vol. 9 [ 2015 ], No. 1 , Art. 7 


Preferences 

Of our 34 respondents, 22 reported preferring handwritten 
feedback, 10 electronic, and 2 indicated no preference. However, 
a significant chi square test for independence revealed that this 
preference varied as a function of the instructor, with one 
instructor's students slightly preferring electronic, whereas 
students in the other two classes preferred handwritten, /(2, N 
= 34) = 10.18, p < .04 (table 1). Another significant chi square 
analysis indicated that students' reported preferences were most 
likely to match that of the form of feedback they actually 
received, j 2 (2, N = 34) = 17.08, p < .01 (table 2). 
Self-Reported Attention to Feedback 

When asked whether they had read their professors' 
feedback, 29 (88 valid percent; 1 student omitted this item) 
claimed to have read between neutral (3) and very carefully (5). 
For the remainder of our analyses we omitted a student who did 
not respond to this item as well as those who reported not 
having read their feedback at all, leaving a sample of 30 
students. A significant Pearson correlation analysis indicated that 
the care with which students reported reading the feedback was 
positively associated with the Quantity, r(28) = .31, p < .05 (1- 
tailed). Similarly, reading care was positively correlated with 
Encoding, r(28) = .33, p < .04 (1-tailed), but not Student Total, 
Correctly Reported Comments, or Accuracy (Quality). 

An independent-samples t-test indicated that students who 
prefer handwritten feedback reported reading their feedback 
marginally more carefully (M = 4.20, SD = .89) than did those 
who prefer electronic feedback (M = 3.50, SD = .93), regardless 
of whether they actually received handwritten feedback, t( 26) = 
1.85, p = .08. 

Feedback Preferences and Students' Recall 

An independent-samples t-test revealed that students who 
prefer handwritten feedback (M = .33, SD = .28) reported a 
higher proportion of the total feedback (Quantity) than did those 
who prefer electronic feedback (M = .17, SD = .08), t( 24) = 
2.46, p = .02 (adjusted for heteroscedasticity). Another 
independent samples t-test revealed that students who prefer 
handwritten feedback (M = .28, SD = .29) had a marginally 
higher Encoding than those who prefer electronic feedback (M = 
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.14, SD = .05), t(22) = 2.02, p < .06 (adjusted for 
heteroscedasticity). 

Effects of Feedback Received 

An independent-samples t-test revealed that students who 
received electronic feedback (M = .77, SD = .23) were 
marginally less accurate in their recall of the feedback they 
received (that is, they had a higher Encoding) than were those 
who received handwritten feedback (M = .91, SD = .14), t( 27) = 
-2.04, p < .06 (adjusted for heteroscedasticity). Another 
independent-samples t-test indicated that students who received 
handwritten feedback (M = .35, SD = .29) had a higher Quantity 
score than those who received electronic feedback (M = .18, SD 
= .09), t(20) = 2.26, p < .04 (adjusted for heteroscedasticity). 
This result may indicate a perception of more feedback received 
in handwritten form, as it includes comments remembered but 
not actually received. 

Discussion 

The present study examines the relationship between 
mode of feedback on student papers and attention to such 
feedback, as well as the impact of that feedback. Results showed 
that although students generally prefer handwritten to electronic 
feedback, they tend to prefer the mode of feedback that they 
actually received from their professors. Not surprisingly, the 
more carefully students read the feedback, the more they 
encoded the feedback (as measured by the ratio of Accurately 
Reported Comments to Instructor Total). Further, students who 
received (or preferred) electronic feedback were more accurate 
in recalling comments received from the instructor, which may 
indicate students more successfully attend to electronic 
feedback. However, those who received (or preferred) 
handwritten feedback recalled a higher quantity of comments 
overall (including inaccurate ones) than did those who received 
electronic feedback. 

Although electronic feedback was slightly more accurately 
encoded in our sample, it appears that format perse does not 
greatly impact student consumption or retention. In fact, in line 
with the work of Gibbs (2002) and Adams (2011), careful 
reading appears to be the key component to retention of 
feedback, regardless of format. Therefore, instructors should 
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take advantage of the strengths of either format to ensure that 
students attend to feedback. For instance, students are more 
likely to respond to feedback if it includes descriptive information 
that specifies the relationship between student performance and 
established assessment criteria (Nicol & Milligan, 2006). 

In our sample, students tended to prefer whatever 
feedback they received, regardless of mechanism. Recall that 
student preferences regarding feedback mechanism have been 
inconsistent across studies. Some students preferred electronic 
feedback because they saw it as easier to read and to reference 
at a later time, while other students expressed concern over 
difficulties with the e-feedback software and/or personal 
computing software (Thomson, 2008). Bridge and Appleyard 
(2008) found that students generally preferred online feedback 
for similar reasons, whereas Brodie and Loch (2009) found that 
students preferred more personal handwritten comments. Our 
research suggests that preferences do not play a vital role in 
formative assessment. Instead, instructors should be consistent 
with their feedback mechanisms. Students are more likely to 
embrace e-feedback when it is used consistently across all or 
nearly all course units (Adams, 2011; Siebert, 2009). This allows 
students to gain a sense of familiarity with the expectations and 
gain the level of digital competency need to encode the 
feedback. 

As is common in pedagogical research, our study has some 
notable limitations. We ended up with a small sample size for 
each of the format groups. This was partially a result of some 
students missing the day we handed out the feedback and 
others missing the day we handed out the survey. Once we 
debriefed the class, we could not collect additional data. The 
small sample size makes the variability in instructors' feedback 
and assignments problematic in terms of interpreting our data. 
However, we argue that future researchers should maintain this 
variability in that it increases generalizability across disciplines 
and instructors. 

In the present study, we attempted to go beyond 
measuring student attitudes about feedback to examine student 
retention of feedback based on whether it is in electronic or 
handwritten format. Although we did not uncover dramatic 
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differences between these two formats, we did find support for 
sticking with a single format so that students know what to 
expect. We therefore encourage instructors to work with 
whatever format fits their teaching style while closely attending 
to the accepted standards for good feedback practice. In addition 
we encourage future researchers to overcome some of our 
limitations in this study and continue to explore this area. 
Technological tools for assessment are growing exponentially, 
and it will benefit all instructors to examine their effects on 
learning. 
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